ebab7d0da53c51438720d60c6e2e467c - /page/wikibook/entry.mu

<!DOCTYPE html>

<title>Value function</title>

</head>

Value function

</h1>

</header>

<div id="mw-content-text" class="mw-body-content mw-content-ltr" lang="en" dir="ltr"><div class="mw-content-ltr mw-parser-output" lang="en" dir="ltr">The value function of an <a href="Optimization_problem" title="Optimization problem">optimization problem</a> gives the <a href="Value_(mathematics)" title="Value (mathematics)">value</a> attained by the <a href="Objective_function" class="mw-redirect" title="Objective function">objective function</a> at a solution, while only depending on the <a href="Parameter" title="Parameter">parameters</a> of the problem.<a href="#cite_note-1">[1]</a><a href="#cite_note-2">[2]</a> In a <a href="Control_theory" title="Control theory">controlled</a> <a href="Dynamical_system" title="Dynamical system">dynamical system</a>, the value function represents the optimal payoff of the system over the interval <var>[t, t1]</var> when started at the time-<var>t</var> <a href="State_variable" title="State variable">state variable</a> <var>x(t)=x</var>.<a href="#cite_note-3">[3]</a> If the objective function represents some cost that is to be minimized, the value function can be interpreted as the cost to finish the optimal program, and is thus referred to as "cost-to-go function."<a href="#cite_note-Bertsekas_Tsitsiklis-4">[4]</a><a href="#cite_note-5">[5]</a> In an economic context, where the objective function usually represents <a href="Utility" title="Utility">utility</a>, the value function is conceptually equivalent to the <a href="Indirect_utility_function" title="Indirect utility function">indirect utility function</a>.<a href="#cite_note-6">[6]</a><a href="#cite_note-7">[7]</a>

In a problem of <a href="Optimal_control" title="Optimal control">optimal control</a>, the value function is defined as the <a href="Supremum" class="mw-redirect" title="Supremum">supremum</a> of the objective function taken over the set of admissible controls. Given <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle (t_{0},x_{0})\in [0,t_{1}]\times \mathbb {R} ^{d}}">

<msub>

</mrow>

</msub>

<msub>

</mrow>

</msub>

<msub>

</mrow>

</msub>

<msup>

</mrow>

</mrow>

</msup>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle (t_{0},x_{0})\in [0,t_{1}]\times \mathbb {R} ^{d}}</annotation>

</semantics>

</math><img src="./61b7207b5b16cedc34a560f756549722cedc590c.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.838ex; width:20.956ex; height:3.176ex;" alt="{\displaystyle (t_{0},x_{0})\in [0,t_{1}]\times \mathbb {R} ^{d}}" loading="lazy">, a typical optimal control problem is to

<mtext>maximize</mtext>

</mrow>

<msub>

</mrow>

</msub>

<msub>

</mrow>

</msub>

<msub>

</mrow>

</msub>

</mrow>

<msub>

</mrow>

</msub>

</mrow>

</msubsup>

</mrow>

<msub>

</mrow>

</msub>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle {\text{maximize}}\quad J(t_{0},x_{0};u)=\int _{t_{0}}^{t_{1}}I(t,x(t),u(t))\,\mathrm {d} t+\phi (x(t_{1}))}</annotation>

</semantics>

</math><img src="./90fe040aca66097527f49271d846de3a50e0ed3c.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -2.671ex; width:57.939ex; height:6.509ex;" alt="{\displaystyle {\text{maximize}}\quad J(t_{0},x_{0};u)=\int _{t_{0}}^{t_{1}}I(t,x(t),u(t))\,\mathrm {d} t+\phi (x(t_{1}))}" loading="lazy"></dd></dl>

subject to

<mfrac>

<mrow>

</mrow>

</mrow>

<mrow>

</mrow>

</mrow>

</mfrac>

</mrow>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle {\frac {\mathrm {d} x(t)}{\mathrm {d} t}}=f(t,x(t),u(t))}</annotation>

</semantics>

</math><img src="./9a4bee21eb12cb803784b6f00916235308d33a8f.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -2.005ex; width:23.158ex; height:5.843ex;" alt="{\displaystyle {\frac {\mathrm {d} x(t)}{\mathrm {d} t}}=f(t,x(t),u(t))}" loading="lazy"></dd></dl>

with initial state variable <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle x(t_{0})=x_{0}}">

<msub>

</mrow>

</msub>

<msub>

</mrow>

</msub>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle x(t_{0})=x_{0}}</annotation>

</semantics>

</math><img src="./aabccfdf83c824bd305a2b9d7554bfc7d30e4763.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.838ex; width:10.515ex; height:2.843ex;" alt="{\displaystyle x(t_{0})=x_{0}}" loading="lazy">.<a href="#cite_note-8">[8]</a> The objective function <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle J(t_{0},x_{0};u)}">

<msub>

</mrow>

</msub>

<msub>

</mrow>

</msub>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle J(t_{0},x_{0};u)}</annotation>

</semantics>

</math><img src="./d581e42aa324dbe406842292331e785f42d561c1.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.838ex; width:10.956ex; height:2.843ex;" alt="{\displaystyle J(t_{0},x_{0};u)}" loading="lazy"> is to be maximized over all admissible controls <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle u\in U[t_{0},t_{1}]}">

<msub>

</mrow>

</msub>

<msub>

</mrow>

</msub>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle u\in U[t_{0},t_{1}]}</annotation>

</semantics>

</math><img src="./81b991151ca37b1da4a5b43ddd0a100a1a25e83f.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.838ex; width:12.068ex; height:2.843ex;" alt="{\displaystyle u\in U[t_{0},t_{1}]}" loading="lazy">, where <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle u}">

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle u}</annotation>

</semantics>

</math><img src="./c3e6bb763d22c20916ed4f0bb6bd49d7470cffd8.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.338ex; width:1.33ex; height:1.676ex;" alt="{\displaystyle u}" loading="lazy"> is a <a href="Measurable_function" title="Measurable function">Lebesgue measurable function</a> from <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle [t_{0},t_{1}]}">

<msub>

</mrow>

</msub>

<msub>

</mrow>

</msub>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle [t_{0},t_{1}]}</annotation>

</semantics>

</math><img src="./ffe2ab6560fe2acf9a63ad878ad482164b79012d.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.838ex; width:6.115ex; height:2.843ex;" alt="{\displaystyle [t_{0},t_{1}]}" loading="lazy"> to some prescribed arbitrary set in <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle \mathbb {R} ^{m}}">

<msup>

</mrow>

</mrow>

</msup>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle \mathbb {R} ^{m}}</annotation>

</semantics>

</math><img src="./6a87a024931038d1858dc22e8a194e5978c3412e.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.338ex; width:3.353ex; height:2.343ex;" alt="{\displaystyle \mathbb {R} ^{m}}" loading="lazy">. The value function is then defined as

</mrow>

</munder>

</mrow>

<msub>

</mrow>

</msub>

</mrow>

</msubsup>

</mrow>

<msub>

</mrow>

</msub>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle V(t,x(t))=\max _{u\in U}\int _{t}^{t_{1}}I(\tau ,x(\tau ),u(\tau ))\,\mathrm {d} \tau +\phi (x(t_{1}))}</annotation>

</semantics>

</math><img src="./1ee2190a68d5d0dad73f3e3d29c1850fff1ca874.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -2.338ex; width:50.651ex; height:6.176ex;" alt="{\displaystyle V(t,x(t))=\max _{u\in U}\int _{t}^{t_{1}}I(\tau ,x(\tau ),u(\tau ))\,\mathrm {d} \tau +\phi (x(t_{1}))}" loading="lazy">

</div>

with <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle V(t_{1},x(t_{1}))=\phi (x(t_{1}))}">

<msub>

</mrow>

</msub>

<msub>

</mrow>

</msub>

<msub>

</mrow>

</msub>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle V(t_{1},x(t_{1}))=\phi (x(t_{1}))}</annotation>

</semantics>

</math><img src="./cfb6084d95a23159a0155f365a5e7be95a1b2e4e.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.838ex; width:22.883ex; height:2.843ex;" alt="{\displaystyle V(t_{1},x(t_{1}))=\phi (x(t_{1}))}" loading="lazy">, where <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle \phi (x(t_{1}))}">

<msub>

</mrow>

</msub>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle \phi (x(t_{1}))}</annotation>

</semantics>

</math><img src="./ac32734f19d7b2c380f26fed299c9624b5910876.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.838ex; width:8.228ex; height:2.843ex;" alt="{\displaystyle \phi (x(t_{1}))}" loading="lazy"> is the "scrap value". If the optimal pair of control and state trajectories is <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle (x^{\ast },u^{\ast })}">

<msup>

</mrow>

</msup>

<msup>

</mrow>

</msup>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle (x^{\ast },u^{\ast })}</annotation>

</semantics>

</math><img src="./a79ef166e63e0531e1a90a0bcb9f8144f0bc8862.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.838ex; width:7.611ex; height:2.843ex;" alt="{\displaystyle (x^{\ast },u^{\ast })}" loading="lazy">, then <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle V(t_{0},x_{0})=J(t_{0},x_{0};u^{\ast })}">

<msub>

</mrow>

</msub>

<msub>

</mrow>

</msub>

<msub>

</mrow>

</msub>

<msub>

</mrow>

</msub>

<msup>

</mrow>

</msup>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle V(t_{0},x_{0})=J(t_{0},x_{0};u^{\ast })}</annotation>

</semantics>

</math><img src="./eb17bcc9395e520b6262c6a3b7f3b11e5f408027.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.838ex; width:24.017ex; height:2.843ex;" alt="{\displaystyle V(t_{0},x_{0})=J(t_{0},x_{0};u^{\ast })}" loading="lazy">. The function <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle h}">

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle h}</annotation>

</semantics>

</math><img src="./b26be3e694314bc90c3215047e4a2010c6ee184a.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.338ex; width:1.339ex; height:2.176ex;" alt="{\displaystyle h}" loading="lazy"> that gives the optimal control <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle u^{\ast }}">

<msup>

</mrow>

</msup>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle u^{\ast }}</annotation>

</semantics>

</math><img src="./607cbfc908268479511f0a0cb0d13cea148b4ca1.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.338ex; width:2.384ex; height:2.343ex;" alt="{\displaystyle u^{\ast }}" loading="lazy"> based on the current state <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle x}">

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle x}</annotation>

</semantics>

</math><img src="./87f9e315fd7e2ba406057a97300593c4802b53e4.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.338ex; width:1.33ex; height:1.676ex;" alt="{\displaystyle x}" loading="lazy"> is called a feedback control policy,<a href="#cite_note-Bertsekas_Tsitsiklis-4">[4]</a> or simply a policy function.<a href="#cite_note-9">[9]</a>

Bellman's principle of optimality roughly states that any optimal policy at time <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle t}">

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle t}</annotation>

</semantics>

</math><img src="./65658b7b223af9e1acc877d848888ecdb4466560.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.338ex; width:0.84ex; height:2.009ex;" alt="{\displaystyle t}" loading="lazy">, <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle t_{0}\leq t\leq t_{1}}">

<msub>

</mrow>

</msub>

<msub>

</mrow>

</msub>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle t_{0}\leq t\leq t_{1}}</annotation>

</semantics>

</math><img src="./a409858d7eb32c359f8ea1f6d5cd3efa96d22ae1.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.671ex; width:10.824ex; height:2.343ex;" alt="{\displaystyle t_{0}\leq t\leq t_{1}}" loading="lazy"> taking the current state <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle x(t)}">

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle x(t)}</annotation>

</semantics>

</math><img src="./d54c275db3a1e620737b58e143b0818107fa5f5c.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.838ex; width:3.979ex; height:2.843ex;" alt="{\displaystyle x(t)}" loading="lazy"> as "new" initial condition must be optimal for the remaining problem. If the value function happens to be <a href="Differentiable_function" title="Differentiable function">continuously differentiable</a>,<a href="#cite_note-10">[10]</a> this gives rise to an important <a href="Partial_differential_equation" title="Partial differential equation">partial differential equation</a> known as <a href="Hamilton%E2%80%93Jacobi%E2%80%93Bellman_equation" title="Hamilton–Jacobi–Bellman equation">Hamilton–Jacobi–Bellman equation</a>,

<mfrac>

<mrow>

</mrow>

<mrow>

</mrow>

</mfrac>

</mrow>

</mrow>

</munder>

<mrow>

<mrow>

<mfrac>

<mrow>

</mrow>

<mrow>

</mrow>

</mfrac>

</mrow>

</mrow>

</mrow>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle -{\frac {\partial V(t,x)}{\partial t}}=\max _{u}\left\{I(t,x,u)+{\frac {\partial V(t,x)}{\partial x}}f(t,x,u)\right\}}</annotation>

</semantics>

</math><img src="./fd0baa584704c952f10fa20b84fa1b332f9d3540.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -2.505ex; width:51.057ex; height:6.343ex;" alt="{\displaystyle -{\frac {\partial V(t,x)}{\partial t}}=\max _{u}\left\{I(t,x,u)+{\frac {\partial V(t,x)}{\partial x}}f(t,x,u)\right\}}" loading="lazy"></dd></dl>

where the <a href="https://en.wiktionary.org/wiki/maximand" class="extiw external" title="wiktionary:maximand">maximand</a> on the right-hand side can also be re-written as the <a href="Hamiltonian_(control_theory)" title="Hamiltonian (control theory)">Hamiltonian</a>, <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle H\left(t,x,u,\lambda \right)=I(t,x,u)+\lambda (t)f(t,x,u)}">

<mrow>

<mrow>

</mrow>

</mrow>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle H\left(t,x,u,\lambda \right)=I(t,x,u)+\lambda (t)f(t,x,u)}</annotation>

</semantics>

</math><img src="./efd719f7189327cb7a9839ed07dd89f478119f6a.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.838ex; width:39.362ex; height:2.843ex;" alt="{\displaystyle H\left(t,x,u,\lambda \right)=I(t,x,u)+\lambda (t)f(t,x,u)}" loading="lazy">, as

<mfrac>

<mrow>

</mrow>

<mrow>

</mrow>

</mfrac>

</mrow>

</mrow>

</munder>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle -{\frac {\partial V(t,x)}{\partial t}}=\max _{u}H(t,x,u,\lambda )}</annotation>

</semantics>

</math><img src="./62e8837c4ba7b87646c0cd0f528ec00d9322be1d.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -2.005ex; width:30.402ex; height:5.843ex;" alt="{\displaystyle -{\frac {\partial V(t,x)}{\partial t}}=\max _{u}H(t,x,u,\lambda )}" loading="lazy"></dd></dl>

with <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle \partial V(t,x)/\partial x=\lambda (t)}">

</mrow>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle \partial V(t,x)/\partial x=\lambda (t)}</annotation>

</semantics>

</math><img src="./84cf1e7a9e337e6193411a02a2b1a7f669b6bbaf.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.838ex; width:19.031ex; height:2.843ex;" alt="{\displaystyle \partial V(t,x)/\partial x=\lambda (t)}" loading="lazy"> playing the role of the <a href="Costate_variable" class="mw-redirect" title="Costate variable">costate variables</a>.<a href="#cite_note-11">[11]</a> Given this definition, we further have <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle \mathrm {d} \lambda (t)/\mathrm {d} t=\partial ^{2}V(t,x)/\partial x\partial t+\partial ^{2}V(t,x)/\partial x^{2}\cdot f(x)}">

</mrow>

</mrow>

</mrow>

<msup>

</mrow>

</msup>

</mrow>

<msup>

</mrow>

</msup>

</mrow>

<msup>

</mrow>

</msup>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle \mathrm {d} \lambda (t)/\mathrm {d} t=\partial ^{2}V(t,x)/\partial x\partial t+\partial ^{2}V(t,x)/\partial x^{2}\cdot f(x)}</annotation>

</semantics>

</math><img src="./13bfb2366a5d47fff44d57399c1739f5c98892e3.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.838ex; width:49.854ex; height:3.176ex;" alt="{\displaystyle \mathrm {d} \lambda (t)/\mathrm {d} t=\partial ^{2}V(t,x)/\partial x\partial t+\partial ^{2}V(t,x)/\partial x^{2}\cdot f(x)}" loading="lazy">, and after differentiating both sides of the HJB equation with respect to <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle x}">

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle x}</annotation>

</semantics>

</math><img src="./87f9e315fd7e2ba406057a97300593c4802b53e4.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.338ex; width:1.33ex; height:1.676ex;" alt="{\displaystyle x}" loading="lazy">,

<mfrac>

<mrow>

<msup>

</mrow>

</msup>

</mrow>

<mrow>

</mrow>

</mfrac>

</mrow>

<mfrac>

<mrow>

</mrow>

<mrow>

</mrow>

</mfrac>

</mrow>

<mfrac>

<mrow>

<msup>

</mrow>

</msup>

</mrow>

<mrow>

<msup>

</mrow>

</msup>

</mrow>

</mfrac>

</mrow>

<mfrac>

<mrow>

</mrow>

<mrow>

</mrow>

</mfrac>

</mrow>

<mfrac>

<mrow>

</mrow>

<mrow>

</mrow>

</mfrac>

</mrow>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle -{\frac {\partial ^{2}V(t,x)}{\partial t\partial x}}={\frac {\partial I}{\partial x}}+{\frac {\partial ^{2}V(t,x)}{\partial x^{2}}}f(x)+{\frac {\partial V(t,x)}{\partial x}}{\frac {\partial f(x)}{\partial x}}}</annotation>

</semantics>

</math><img src="./016c9bfa88f167af1ba41468e29b705844175e3d.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -2.171ex; width:54.081ex; height:6.176ex;" alt="{\displaystyle -{\frac {\partial ^{2}V(t,x)}{\partial t\partial x}}={\frac {\partial I}{\partial x}}+{\frac {\partial ^{2}V(t,x)}{\partial x^{2}}}f(x)+{\frac {\partial V(t,x)}{\partial x}}{\frac {\partial f(x)}{\partial x}}}" loading="lazy"></dd></dl>

which after replacing the appropriate terms recovers the <a href="Costate_equation" title="Costate equation">costate equation</a>

<mover>

</mover>

</mrow>

<mrow>

<mfrac>

<mrow>

</mrow>

<mrow>

</mrow>

</mfrac>

</mrow>

<mfrac>

<mrow>

</mrow>

<mrow>

</mrow>

</mfrac>

</mrow>

</munder>

</mrow>

<mfrac>

<mrow>

</mrow>

<mrow>

</mrow>

</mfrac>

</mrow>

</munder>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle -{\dot {\lambda }}(t)=\underbrace {{\frac {\partial I}{\partial x}}+\lambda (t){\frac {\partial f(x)}{\partial x}}} _{={\frac {\partial H}{\partial x}}}}</annotation>

</semantics>

</math><img src="./60d45744e471ca9c4787e3c96510178bd6722eab.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -7.171ex; margin-right: -0.028ex; width:25.839ex; height:11.009ex;" alt="{\displaystyle -{\dot {\lambda }}(t)=\underbrace {{\frac {\partial I}{\partial x}}+\lambda (t){\frac {\partial f(x)}{\partial x}}} _{={\frac {\partial H}{\partial x}}}}" loading="lazy"></dd></dl>

where <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle {\dot {\lambda }}(t)}">

<mover>

</mover>

</mrow>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle {\dot {\lambda }}(t)}</annotation>

</semantics>

</math><img src="./0a6fba85c32408779174edc1e0bdc89202ae817b.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.838ex; width:4.004ex; height:3.176ex;" alt="{\displaystyle {\dot {\lambda }}(t)}" loading="lazy"> is <a href="Newton_notation" class="mw-redirect" title="Newton notation">Newton notation</a> for the derivative with respect to time.<a href="#cite_note-12">[12]</a>

The value function is the unique <a href="Viscosity_solution" title="Viscosity solution">viscosity solution</a> to the Hamilton–Jacobi–Bellman equation.<a href="#cite_note-13">[13]</a> In an <a href="Online_algorithm" title="Online algorithm">online</a> closed-loop approximate optimal control, the value function is also a <a href="Lyapunov_function" title="Lyapunov function">Lyapunov function</a> that establishes global asymptotic stability of the closed-loop system.<a href="#cite_note-14">[14]</a>

<div class="mw-heading mw-heading2"><h2 id="References">References</h2></div>

/* start https://en.wikipedia.org/ */

.mw-parser-output .reflist{margin-bottom:0.5em;list-style-type:decimal}@media screen{.mw-parser-output .reflist{font-size:90%}}.mw-parser-output .reflist .references{font-size:100%;margin-bottom:0;list-style-type:inherit}.mw-parser-output .reflist-columns-2{column-width:30em}.mw-parser-output .reflist-columns-3{column-width:25em}.mw-parser-output .reflist-columns{margin-top:0.3em}.mw-parser-output .reflist-columns ol{margin-top:0}.mw-parser-output .reflist-columns li{page-break-inside:avoid;break-inside:avoid-column}.mw-parser-output .reflist-upper-alpha{list-style-type:upper-alpha}.mw-parser-output .reflist-upper-roman{list-style-type:upper-roman}.mw-parser-output .reflist-lower-alpha{list-style-type:lower-alpha}.mw-parser-output .reflist-lower-greek{list-style-type:lower-greek}.mw-parser-output .reflist-lower-roman{list-style-type:lower-roman}

/* end https://en.wikipedia.org/ */

</style><div class="reflist">

/* start https://en.wikipedia.org/ */

.mw-parser-output cite.citation{font-style:inherit;word-wrap:break-word}.mw-parser-output .citation q{quotes:"\"""\"""'""'"}.mw-parser-output .citation:target{background-color:rgba(0,127,255,0.133)}.mw-parser-output .id-lock-free.id-lock-free a{background:url("./mw/Lock-green.svg")right 0.1em center/9px no-repeat}.mw-parser-output .id-lock-limited.id-lock-limited a,.mw-parser-output .id-lock-registration.id-lock-registration a{background:url("./mw/Lock-gray-alt-2.svg")right 0.1em center/9px no-repeat}.mw-parser-output .id-lock-subscription.id-lock-subscription a{background:url("./mw/Lock-red-alt-2.svg")right 0.1em center/9px no-repeat}.mw-parser-output .cs1-ws-icon a{background:url("./mw/Wikisource-logo.svg")right 0.1em center/12px no-repeat}body:not(.skin-timeless):not(.skin-minerva) .mw-parser-output .id-lock-free a,body:not(.skin-timeless):not(.skin-minerva) .mw-parser-output .id-lock-limited a,body:not(.skin-timeless):not(.skin-minerva) .mw-parser-output .id-lock-registration a,body:not(.skin-timeless):not(.skin-minerva) .mw-parser-output .id-lock-subscription a,body:not(.skin-timeless):not(.skin-minerva) .mw-parser-output .cs1-ws-icon a{background-size:contain;padding:0 1em 0 0}.mw-parser-output .cs1-code{color:inherit;background:inherit;border:none;padding:inherit}.mw-parser-output .cs1-hidden-error{display:none;color:var(--color-error,#d33)}.mw-parser-output .cs1-visible-error{color:var(--color-error,#d33)}.mw-parser-output .cs1-maint{display:none;color:#085;margin-left:0.3em}.mw-parser-output .cs1-kern-left{padding-left:0.2em}.mw-parser-output .cs1-kern-right{padding-right:0.2em}.mw-parser-output .citation .mw-selflink{font-weight:inherit}@media screen{.mw-parser-output .cs1-format{font-size:95%}html.skin-theme-clientpref-night .mw-parser-output .cs1-maint{color:#18911f}}@media screen and (prefers-color-scheme:dark){html.skin-theme-clientpref-os .mw-parser-output .cs1-maint{color:#18911f}}

/* end https://en.wikipedia.org/ */

</style><cite id="CITEREFFlemingRishel1975" class="citation book cs1"><a href="Wendell_Fleming" title="Wendell Fleming">Fleming, Wendell H.</a>; Rishel, Raymond W. (1975). <a rel="nofollow" class="external text" href="https://books.google.com/books?id=qJDbBwAAQBAJ&pg=PA81">Deterministic and Stochastic Optimal Control</a>. New York: Springer. pp. 81–83. <a href="ISBN_(identifier)" class="mw-redirect" title="ISBN (identifier)">ISBN</a> <bdi>0-387-90155-8</bdi>.</cite>

</li>

<li id="cite_note-2"><a href="#cite_ref-2">^</a> <cite id="CITEREFCaputo2005" class="citation book cs1">Caputo, Michael R. (2005). <a rel="nofollow" class="external text" href="https://books.google.com/books?id=XZ2yYSVKWJkC&pg=PA185">Foundations of Dynamic Economic Analysis : Optimal Control Theory and Applications</a>. New York: Cambridge University Press. p. 185. <a href="ISBN_(identifier)" class="mw-redirect" title="ISBN (identifier)">ISBN</a> <bdi>0-521-60368-4</bdi>.</cite>

</li>

<li id="cite_note-3"><a href="#cite_ref-3">^</a> <cite id="CITEREFWeber2011" class="citation book cs1">Weber, Thomas A. (2011). Optimal Control Theory : with Applications in Economics. Cambridge: The MIT Press. p. 82. <a href="ISBN_(identifier)" class="mw-redirect" title="ISBN (identifier)">ISBN</a> <bdi>978-0-262-01573-8</bdi>.</cite>

</li>

<li id="cite_note-Bertsekas_Tsitsiklis-4">^ <a href="#cite_ref-Bertsekas_Tsitsiklis_4-0">a</a> <a href="#cite_ref-Bertsekas_Tsitsiklis_4-1">b</a> <cite id="CITEREFBertsekasTsitsiklis1996" class="citation book cs1">Bertsekas, Dimitri P.; Tsitsiklis, John N. (1996). Neuro-Dynamic Programming. Belmont: Athena Scientific. p. 2. <a href="ISBN_(identifier)" class="mw-redirect" title="ISBN (identifier)">ISBN</a> <bdi>1-886529-10-8</bdi>.</cite>

</li>

<li id="cite_note-5"><a href="#cite_ref-5">^</a> <cite class="citation web cs1"><a rel="nofollow" class="external text" href="https://stanford.edu/class/ee365/lectures/dp.pdf#page=3">"EE365: Dynamic Programming"</a> (PDF).</cite>

</li>

<li id="cite_note-6"><a href="#cite_ref-6">^</a> <cite id="CITEREFMas-ColellWhinstonGreen1995" class="citation book cs1"><a href="Andreu_Mas-Colell" title="Andreu Mas-Colell">Mas-Colell, Andreu</a>; <a href="Michael_Whinston" title="Michael Whinston">Whinston, Michael D.</a>; Green, Jerry R. (1995). Microeconomic Theory. New York: Oxford University Press. p. 964. <a href="ISBN_(identifier)" class="mw-redirect" title="ISBN (identifier)">ISBN</a> <bdi>0-19-507340-1</bdi>.</cite>

</li>

<li id="cite_note-7"><a href="#cite_ref-7">^</a> <cite id="CITEREFCorbaeStinchcombeZeman2009" class="citation book cs1">Corbae, Dean; Stinchcombe, Maxwell B.; Zeman, Juraj (2009). <a rel="nofollow" class="external text" href="https://books.google.com/books?id=j5P83LtzVO8C&pg=PA145">An Introduction to Mathematical Analysis for Economic Theory and Econometrics</a>. Princeton University Press. p. 145. <a href="ISBN_(identifier)" class="mw-redirect" title="ISBN (identifier)">ISBN</a> <bdi>978-0-691-11867-3</bdi>.</cite>

</li>

<li id="cite_note-8"><a href="#cite_ref-8">^</a> <cite id="CITEREFKamienSchwartz1991" class="citation book cs1"><a href="Morton_Kamien" title="Morton Kamien">Kamien, Morton I.</a>; Schwartz, Nancy L. (1991). Dynamic Optimization : The Calculus of Variations and Optimal Control in Economics and Management (2nd ed.). Amsterdam: North-Holland. p. 259. <a href="ISBN_(identifier)" class="mw-redirect" title="ISBN (identifier)">ISBN</a> <bdi>0-444-01609-0</bdi>.</cite>

</li>

<li id="cite_note-9"><a href="#cite_ref-9">^</a> <cite id="CITEREFLjungqvistSargent2018" class="citation book cs1"><a href="Lars_Ljungqvist" title="Lars Ljungqvist">Ljungqvist, Lars</a>; <a href="Thomas_J._Sargent" title="Thomas J. Sargent">Sargent, Thomas J.</a> (2018). <a rel="nofollow" class="external text" href="https://books.google.com/books?id=Jm1qDwAAQBAJ&pg=PA106">Recursive Macroeconomic Theory</a> (Fourth ed.). Cambridge: MIT Press. p. 106. <a href="ISBN_(identifier)" class="mw-redirect" title="ISBN (identifier)">ISBN</a> <bdi>978-0-262-03866-9</bdi>.</cite>

</li>

<li id="cite_note-10"><a href="#cite_ref-10">^</a> Benveniste and <a href="Jos%C3%A9_Scheinkman" title="José Scheinkman">Scheinkman</a> established sufficient conditions for the differentiability of the value function, which in turn allows an application of the <a href="Envelope_theorem" title="Envelope theorem">envelope theorem</a>, see <cite id="CITEREFBenvenisteScheinkman1979" class="citation journal cs1">Benveniste, L. M.; Scheinkman, J. A. (1979). "On the Differentiability of the Value Function in Dynamic Models of Economics". Econometrica. 47 (3): 727–732. <a href="Doi_(identifier)" class="mw-redirect" title="Doi (identifier)">doi</a>:<a rel="nofollow" class="external text" href="https://doi.org/10.2307%2F1910417">10.2307/1910417</a>. <a href="JSTOR_(identifier)" class="mw-redirect" title="JSTOR (identifier)">JSTOR</a> <a rel="nofollow" class="external text" href="https://www.jstor.org/stable/1910417">1910417</a>.</cite> Also see <cite id="CITEREFSeierstad1982" class="citation journal cs1">Seierstad, Atle (1982). "Differentiability Properties of the Optimal Value Function in Control Theory". Journal of Economic Dynamics and Control. 4: 303–310. <a href="Doi_(identifier)" class="mw-redirect" title="Doi (identifier)">doi</a>:<a rel="nofollow" class="external text" href="https://doi.org/10.1016%2F0165-1889%2882%2990019-7">10.1016/0165-1889(82)90019-7</a>.</cite>

</li>

<li id="cite_note-11"><a href="#cite_ref-11">^</a> <cite id="CITEREFKirk1970" class="citation book cs1">Kirk, Donald E. (1970). Optimal Control Theory. Englewood Cliffs, NJ: Prentice-Hall. p. 88. <a href="ISBN_(identifier)" class="mw-redirect" title="ISBN (identifier)">ISBN</a> <bdi>0-13-638098-0</bdi>.</cite>

</li>

<li id="cite_note-12"><a href="#cite_ref-12">^</a> <cite id="CITEREFZhou1990" class="citation journal cs1">Zhou, X. Y. (1990). "Maximum Principle, Dynamic Programming, and their Connection in Deterministic Control". Journal of Optimization Theory and Applications. 65 (2): 363–373. <a href="Doi_(identifier)" class="mw-redirect" title="Doi (identifier)">doi</a>:<a rel="nofollow" class="external text" href="https://doi.org/10.1007%2FBF01102352">10.1007/BF01102352</a>. <a href="S2CID_(identifier)" class="mw-redirect" title="S2CID (identifier)">S2CID</a> <a rel="nofollow" class="external text" href="https://api.semanticscholar.org/CorpusID:122333807">122333807</a>.</cite>

</li>

<li id="cite_note-13"><a href="#cite_ref-13">^</a> Theorem 10.1 in <cite id="CITEREFBressan2019" class="citation web cs1">Bressan, Alberto (2019). <a rel="nofollow" class="external text" href="http://personal.psu.edu/axb62/PSPDF/HJlnotes19.pdf#page=54">"Viscosity Solutions of Hamilton-Jacobi Equations and Optimal Control Problems"</a> (PDF). Lecture Notes.</cite>

</li>

<li id="cite_note-14"><a href="#cite_ref-14">^</a> <cite id="CITEREFKamalapurkarWaltersRosenfeldDixon2018" class="citation book cs1">Kamalapurkar, Rushikesh; Walters, Patrick; Rosenfeld, Joel; Dixon, Warren (2018). <a rel="nofollow" class="external text" href="https://books.google.com/books?id=R3haDwAAQBAJ&pg=PA27">"Optimal Control and Lyapunov Stability"</a>. Reinforcement Learning for Optimal Feedback Control: A Lyapunov-Based Approach. Berlin: Springer. pp. 26–27. <a href="ISBN_(identifier)" class="mw-redirect" title="ISBN (identifier)">ISBN</a> <bdi>978-3-319-78383-3</bdi>.</cite>

</li>

</ol></div></div>

<div class="mw-heading mw-heading2"><h2 id="Further_reading">Further reading</h2></div>

<ul><li><cite id="CITEREFCaputo2005" class="citation book cs1">Caputo, Michael R. (2005). <a rel="nofollow" class="external text" href="https://books.google.com/books?id=XZ2yYSVKWJkC&pg=PA174">"Necessary and Sufficient Conditions for Isoperimetric Problems"</a>. Foundations of Dynamic Economic Analysis : Optimal Control Theory and Applications. New York: Cambridge University Press. pp. 174–210. <a href="ISBN_(identifier)" class="mw-redirect" title="ISBN (identifier)">ISBN</a> <bdi>0-521-60368-4</bdi>.</cite></li>

<li><cite id="CITEREFClarkeLoewen1986" class="citation journal cs1">Clarke, Frank H.; Loewen, Philip D. (1986). "The Value Function in Optimal Control: Sensitivity, Controllability, and Time-Optimality". SIAM Journal on Control and Optimization. 24 (2): 243–263. <a href="Doi_(identifier)" class="mw-redirect" title="Doi (identifier)">doi</a>:<a rel="nofollow" class="external text" href="https://doi.org/10.1137%2F0324014">10.1137/0324014</a>.</cite></li>

<li><cite id="CITEREFLaFranceBarney1991" class="citation journal cs1">LaFrance, Jeffrey T.; Barney, L. Dwayne (1991). <a rel="nofollow" class="external text" href="http://ageconsearch.umn.edu/record/259398/files/agecon-montanastate-003.pdf">"The Envelope Theorem in Dynamic Optimization"</a> (PDF). Journal of Economic Dynamics and Control. 15 (2): 355–385. <a href="Doi_(identifier)" class="mw-redirect" title="Doi (identifier)">doi</a>:<a rel="nofollow" class="external text" href="https://doi.org/10.1016%2F0165-1889%2891%2990018-V">10.1016/0165-1889(91)90018-V</a>.</cite></li>

<li><cite id="CITEREFStengel1994" class="citation book cs1">Stengel, Robert F. (1994). <a rel="nofollow" class="external text" href="https://books.google.com/books?id=jDjPxqm7Lw0C&pg=PA201">"Conditions for Optimality"</a>. Optimal Control and Estimation. New York: Dover. pp. 201–222. <a href="ISBN_(identifier)" class="mw-redirect" title="ISBN (identifier)">ISBN</a> <bdi>0-486-68200-5</bdi>.</cite></li></ul></div><div><div class="zim-footer">

This article is issued from <a class="external text" title="Last edited on 2023-07-31" href="https://en.wikipedia.org/wiki/?title=Value_function&oldid=1168085806">Wikipedia</a>. The text is available under <a class="external text" href="https://creativecommons.org/licenses/by-sa/4.0/deed.en">Creative Commons Attribution-Share Alike 4.0</a> unless otherwise noted. Additional terms may apply for the media files.

</div>

</div></div>

</div>

</main>

</div>

</body></html>